lec'dPCT/PTO, 28 SEP 2004 
10 / 50945? 

PCT/IB03/01183 

1 

Method of determining an image from an image sequence 



WO 03/083777 



The invention relates to a method of determining a corresponding image for a 
reference image from an image sequence of a moving object, the image sequence 
representing the object motion as a sequence of states of motion. The invention also relates to 
a system and to an examination apparatus whereby the method can be carried out as well as 
5 to a computer program and a computer program product enabling a data processing unit to 
carry out the method. 



The method is used, for example, wherever an image of a state of motion is to 

10 be determined in an image sequence, said state of motion also being represented in a 

reference image acquired during a second, similar motion of the object. While the object 
performs the motions, each time a signal is determined which represents the sequence in time 
of the states of motion for each motion. Apparatus of this kind are known, for example, from 
the medical field. The patent document US 4,729,379 discloses an X-ray examination 

1 5 apparatus for cardiological examinations in which two X-ray image sequences of the beating 
heart are acquired. A contrast medium is injected into the blood vessels of the heart during 
the acquisition of one of the two sequences. 

During a subsequent step, the two sequences are subtracted from one another, 
that is, one image after the other, so that only the vessels filled with the contrast medium are 

20 reproduced with a minimum amount of background in the resultant differential sequence. To 
this end, the two image sequences must be aligned relative to one another in such a manner 
that from each image sequence always those images are subtracted which represent the same 
state of motion. This is achieved by the acquisition of a respective electrocardiographic 
(ECG) signal by means of an electrocardiograph, that is, in parallel with the acquisition of the 

25 two image sequences. In both ECGs each time two successive R deflections are determined 
whereby the two ECGs are aligned relative to one another. If the time elapsing between the 
two R deflections in the two ECGs differs, this time difference is compensated by linear 
interpolation, so that the images acquired between the R deflections of the two associated 
image sequences can be associated with one another. 
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Because only one reference instant is used for cardiac cycle, the image 
sequences are aligned with one another at one instant only, so that differences between the 
two ECG signals in respect of the duration of the overall cardiac cycle on the one hand and in 
respect of the expansion or compression of individual segments of the motion of the heart on 
5 the other hand are not taken into account. This gives rise to undesirable and disturbing 
artefacts in the differential sequence. 



Therefore, it is an object of the present invention to improve systems of the 

10 kind set forth. 

The object is achieved in accordance with the invention by means of a method 
of determining a corresponding image for a reference image from an image sequence of a 
moving object by means of a first and a second motion signal, in which 

- the first and the second motion signal represent the respective variation in time of the 
1 5 states of motion of a first motion and a second motion of the object, 

- the image sequence represents the first motion of the object as a sequence of images of 
states of motion, 

- the reference image represents a state of motion from the second object motion and is 
acquired at a reference instant during the second motion of the object, 

20 including the following steps: 

a. determining a similarity function by way of a similarity comparison of the first and 
the second motion signal, 

b. determining a correspondence instant in the first motion signal by means of the 
similarity function, the correspondence instant corresponding to the acquisition 

25 instant of the reference image from the second motion signal, 

c. determining, using the first motion signal, that image of the image sequence whose 
acquisition instant corresponds at least approximately to the correspondence instant. 

The above method serves to determine a corresponding image for a reference 
30 image, the corresponding image representing at least approximately that state of motion of a 
moving object which is represented in a reference image. From a first motion of an object 
there is now acquired an image sequence in which each image represents a state of motion of 
the object motion. The succession of images then represents a motion image sequence of the 
object motion. While the object performs a second motion, a reference image is acquired of a 
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state of motion which occurs during the second motion of the object. A motion signal which 
characterizes or represents the variation in time of the states of motion of the motion is 
available from the first as well as from the second motion. A signal of this kind is, for 
example, an ECG which can be acquired while the relevant motion takes place. Another 
5 motion signal is a signal produced by a breathing sensor during the respiratory motion of a 
patient. 

During a first step of the method the two motion signals are examined for 
similarities. This yields a similarity function which can be used to associate with any instant 
of one motion signal a corresponding instant of the other motion signal in such a manner that 

10 the object has assumed at least approximately the same state of motion at the two instants. 
Using the similarity function, in two motions there can be determined two instants at which 
the object has assumed approximately the same state of motion during the respective 
motions. Even when the motions differ to such an extent that the motion signals of the 
motions are non-linearly distorted relative to one another, as opposed to known methods, the 

1 5 method in accordance with the invention still produces results that are suitable for evaluation. 

During a second step of the method the instant in the first motion signal which 
corresponds to the reference instant of the second motion signal is determined. In a third step 
of the method, that image of the image sequence whose acquisition instant corresponds 
approximately to the corresponding instant is determined as the corresponding image. The 

20 corresponding image thus selected represents at least approximately that state of motion of 
the moving object which is represented in the reference image. When the reference image 
and the corresponding image from the image sequence are subtracted from one another, the 
subtraction image will exhibit only a very small number of artifacts which may be due to the 
fact that two images of different states of motion are subtracted from one another. 

25 In conformity with claim 2, the similarity comparison can be performed by 

means of the known "dynamic time warping" method. This method enables a very fast and 
efficient execution of the similarity comparison. 

If the object motion is known in principle, so that additionally information can 
be provided on the states of motion assumed by the object between the acquisitions of the 

30 images of the image sequence, artificial intermediate images can be formed for these states of 
motion by interpolation. For example, the motions performed by some organs during 
respiration can be sufficiently accurately described by means of a motion model. In 
conformity with claim 3 it is then possible to form an image which represents as well as 
possible the state of motion assumed by the object during the acquisition of the reference 
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image. This is advantageous notably when the difference between the actual instants of 
acquisition of the images of the image sequence and the correspondence instant is so large 
that too many artefacts are produced, for example, in a subtraction image. The interpolation 
of intermediate images can also be advantageously used when only few images can be 
5 acquired for the image sequence during the object motion. 

In conformity with claim 4 the method can be used particularly 
advantageously in systems in which images and image sequences of a human or animal heart 
are formed and, moreover, an ECG of the cardiac motion is available. The method in 
accordance with the invention provides reliable determination of images representing the 

10 same state of motion of the heart notably in the case of patients who, because of disease or 
given physical conditions, have cardiac cycles whose ECGs exhibit non-linear distortions 
relative to one another. 

In conformity with claim 5, the method is used in systems in which subtraction 
angiography is carried out. Imaging methods which are suitable for the acquisition of images 

15 of states of motion of the heart may be X-ray systems in conformity with claim 6 and 

ultrasound systems in conformity with claim 7. However, imaging systems which produce 
slice images or volume images, such as magnetic resonance tomography apparatus or X-ray 
computed tomography apparatus, will in the very near future also be capable of recording 
states of motion of the heart. The method can then be used accordingly. 

20 Therefore, in conformity with claim 8 it is particularly advantageous to use the 

method in a system which comprises a data processing unit of the kind used in contemporary 
imaging systems in the medical field. An X-ray examination system may be provided with 
such a system in accordance with claim 9. If the data processing unit is constructed so as to 
be programmable, a computer program or computer program product as claimed in claim 10 

25 can enable the data processing unit to carry out the method in accordance with the invention. 

Embodiments of the invention will be described in detail hereinafter with 
reference to the Figures. 



30 Fig. 1 is a diagrammatic representation of two ECGs in combination with 

image acquisitions. 

Fig. 2 shows a general method for the similarity comparison of two signals. 
Fig. 3 shows the comparable task setting in speech recognition. 
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Fig. 4 shows, by way of example, local and cumulative distances as well as the 
relevant recursion matrix. 

Fig. 5 and Fig. 6 show respective diagrams of some local distances for 
different ECGs. 

5 Fig. 7 illustrates the determination of a correspondence image. 

Fig. 8 shows an X-ray examination system. 



Fig. 1 is a diagrammatic representation of an electrocardiogram (ECG) El of a 
10 cardiac cycle over the time t. Electrocardiography is a method of recording the action 

currents of the human or animal heart. The excited cardiac muscle location has an electric 
charge relative to the non-excited location; this charge propagates at a given speed through 
the remaining segments of the cardiac muscle. Such currents can be measured in time in 
known manner by means of suitable electrodes attached to the body and can be reproduced in 
1 5 conformity with Fig. 1 . The typical duration of a human cardiac cycle is approximately one 
second. 

During the cardiac cycle respective images II to 114 of the heart are acquired 
at a plurality of instants tl to tl4. Each of the images II to 114 represents a state of motion of 
the heart as an instantaneous image of the very complex cardiac motion. Images of this kind 

20 can be acquired by means of known imaging methods, for example, by means of X-ray 
imaging or ultrasound imaging. Contemporary X-ray systems enable the acquisition of a 
maximum of from 30 to 60 images per second, so that the images of the image sequence of a 
cardiac cycle represent 30 and 60 different states of motion of the heart, respectively. Such a 
number of images, however, is difficult to represent in the Figures, so that a smaller number 

25 is used herein. 

For some examinations of the heart it is useful to image only the vascular tree 
of the blood vessels of the heart. To this end, a first and a second image are acquired of a 
state of motion of the heart. During the acquisition of the second image, for example, a 
contrast medium which absorbs X-rays is introduced into the blood vessels of the heart, for 
30 example, by means of a catheter, so that the blood vessels are highlighted very well in the X- 
ray system. The acquisition of the first image takes place without contrast medium. During a 
next step, both images are subtracted from one another, for example, one pixel after the other, 
so that in the ideal case only the vascular tree filled with contrast medium is still visible. This 
method is also referred to as subtraction angiography; in the case of digital images it is also 
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called digital subtraction angiography (DSA). Fig. 1 illustrates diagrarnmatically the 
formation of a second image sequence during which the cardiac vessels are filled with a 
contrast medium, as well as the corresponding ECG El' of the cardiac cycle. The images I'l 
to 1*14 are acquired at the instants t'l to t'14. 
5 An image produced by DSA contains particularly few artefacts when the 

image elements or pixels to be removed by the subtraction are substantially the same in the 
two images. Furthermore, the shape of the object to be highlighted in the two images should 
also be substantially the same. Therefore, notably in the case of a complex motion such as the 
motion of the heart it is important to subtract two images from one another which represent 

10 the same state of motion of the heart as well as possible. For example, if the state of motion 
represented in the image 18 is to be represented as a DSA image, it is necessary to find from 
the second image sequence the image 1*8 which represents substantially the same state of 
motion of the heart as the image 18. This is denoted by the dotted line. If, like in systems used 
thus far, exclusively the so-called R deflection (pronounced deflection briefly before the 

1 5 instant of acquisition of 18) is taken as a reference instant and if this deflection is 

substantially wider in the ECG E'l than in the ECG El (dashed part of the ECG El'), a 
known method would produce an image I'8 which is still situated in the descending part of 
the R deflection, so that it represents a state of motion completely other than the actually 
searched 1*8. Notably in the case of patients suffering from cardiac disease it cannot be 

20 assumed simply that the curve of the ECG of two different cardiac cycles is exactly the same. 
Generally speaking, one ECG exhibits non-linear distortions relative to the other ECG, so 
that at a given instant during the acquisition of one ECG the heart is in a state of motion 
which differs from that during the acquisition of the other ECG. This makes the comparison 
of two ECGs for the determination of the image I'8 corresponding to the image 18 more 

25 difficult. 

Fig. 2 is a general representation of a feasible method enabling two temporally 
non-linearly distorted signals to be examined in respect of similarity. The (one-dimensional) 
functions, plotted horizontally and vertically, correspond to two approximately the same 
functions. The different temporal structuring of the two functions becomes manifest in the 
30 relative distortion of their time scale. A supposed association of instants of corresponding 
events, such as states of motion, is marked as a path in the product grid of the scale. 

The use of such pattern comparison algorithms is known from word 
recognition systems utilized in the field of speech recognition (for example, from the book by 
E. G. Schukat-Talamazzini "Automatische Spracherkennung", ISBN 3-528-05492-1, Vieweg 
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Verlag, 1995, Chapter 5.1). In conformity with Fig. 3 there is given a vocabulary Y = Y V .Y T , 
each entry of the vocabulary being represented by a reference pattern Y t (also called a 
prototype or template) and each reference pattern Y t being given in the form of a feature 
vector sequence Y t = y n ...y^ . A vocabulary entry forms a selected realization of the words 

5 which have been acquired during the system dimensioning phase. Furthermore, a short-term 
analyzed word is given in the form of the feature vector sequence X = x x ...x r . The task of 
recognizing individual words consists in determining the identity of the supposedly spoken 
word from the vocabulary by examining which word of the vocabulary corresponds best to 
the actually spoken word. 

10 The distance D(X y Y) between the input sequence X = x x .. jc t and a reference 

sequence Y = y x ...y s of different duration S * T is determined as the sum of local distances 
d y = d(x t> yj) along a suitable time warping path between the vector sequences. The local 
distance function d(- 9 ) is realized, for example, by the Euclidean metric. The appropriate 
distortion function for this purpose should map X on Y over its entire length, satisfy given 

1 5 properties of monotony and constancy in the t scale and the s scale and yield the smallest 
overall distance. This very complex discrete optimization task (the number of combinatorily 
feasible paths increases exponentially as a function of the length of the prototype even when 
said limitations are imposed) satisfies the principle of optimization and hence can be solved 
by means of the generally known "dynamic programming" method. The cumulative distances 

20 D tJ = D(x x ...x, , y x ^y } ) between initial segments of the vector sequences X , Y are subject to 
the recursion formulas: 



25 out merely 0(T • S) arithmetic operations. Fig. 4 shows the matrices of the local distances 

LD and the cumulative distances KD for a simple example. The optimum distortion path, that 
is, the path with a minimum distance, is highlighted by the squares enclosed by heavy lines. 
It can be determined by the setting of pointers in a recursion matrix RV in conformity with 
the above minimum decision in the recursion formula, that is, simultaneously with the overall 

30 distance. The number and shape of the terms in the minimum expression of the recursion 




Therefore, the overall distance D(X,Y) = £> ra can be determined by carrying 
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formula are dependent on the permitted local transitions LT of the distortion function. In 
addition to the structure chosen in the example, many other modified path restrictions are 
known. This known algorithm for the time-elastic distance calculation is referred to as 
dynamic time warping (DTW). 
5 This method enables two ECG curves to be compared for similarities. To this 

end, the ECG curves are presented in digital form, that is, represented by a large number of 
characteristic points (for example, 300) in each curve; this can be realized in known manner 
by sampling the ECG signal. The ECG curves thus form a sequence of scalars. The local 
distance function can then be expressed by the following alternative standards: 

10 d g = d(x g9 yj) = \x t -yj\ 9 

d lj ^d{x n y J )^{x i -y J f 9 ox 

generally df=d(x i% yj) = ]x„y\ with \x 9 y\ as metric. 

The expression \x,y\ is generally known from mathematics and satisfies the 
conditions ||x,j;|| + ||y,z||>||x,z|| (triangular inequality), ||x,x|| = 0, |x,^|^0 and |x,j>I = |y,*| 
15 (symmetry). 

As before, the cumulative distances are formed as the sum of the preceding 
local distances and the most favorable distortion path is determined recursively with the 
minimum condition. Once the distortion path has been defined, a corresponding characteristic 
point of the other ECG curve can be indicated for each characteristic point of one ECG curve. 
20 During the determination of the optimum path the recursion formula defines 

which boxes can be reached in a recursion step. In accordance with the above formula, the 
minimum is determined from three different cumulative distances, the three distances being 
measured from one box to directly neighboring boxes. Alternatively, the following recursion 
matrix, resulting from the associated recursion formula, can be used: 

0 /=y=o 

min{A j. x , A-w-i • A-aj-i 1 + d v i>0 >J >0 



25 D 9 



oo else 



In this case three cumulative distances are compared again, be it that the 
corresponding boxes are not direct neighbors in one of the three comparisons. The distances 
that are actually compared with one another in the recursion formula are dependent on the 
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exact system conditions and can be varied, if desired, in conformity with the predetermined 
task by the back-pointers or local transitions given by the recursion formula. 

The Figs. 5 and 6 show a respective diagram of some local distances for two 
ECGs. For the sake of simplification, the signal values are rounded off (quantized) to the 
5 corresponding integer values with absolute values of between 0 and 1 0. There is a box for 
each characteristic point of the time axes t and the first characteristic points are denoted each 
time by xl to x5 and yl to y5. The local distances are calculated each time as 
d y = d(x t9 yf) = \x t - yj | . The optimum path is determined by means of the cumulative 

distances (not shown here) and the recursion formula 

0 i=/=0 

10 D y =^min{i) M ^ 1 ,i) M j,Aj-i}+^ *>0,j>0 

oo else 

The two ECG curves E2 and E3 are substantially the same in Fig. 5, so that the optimum 
path, represented by boxes enclosed by heavy lines, extends as a diagonal. The ECG E4 in 
Fig. 6 exhibits slight non-linear distortions relative to the ECG E2. In zones in which the 
course of the two curves is approximately the same, the optimum path is practically diagonal. 

15 In zones in which the paths of the signal deviate, the optimum path deviates from the 

diagonal. When the optimum path has been defined, a corresponding instant x of the ECG E2 
can be determined for each instant y of the ECG E4 and vice versa, the ECG curves then 
representing substantially the same state of motion of the heart at corresponding instants. 

When the corresponding instants have been defined, Fig. 7 shows possibilities 

20 for selecting corresponding images in conformity therewith. During the acquisition of the 
ECG E5, the vessels of the heart are filled with a contrast medium and the image 151 is 
acquired at an instant t51. No contrast medium was used during the acquisition of the ECGs 
E6 and E7 and the associated image sequences. The similarity comparison with the ECG E6 
yields the corresponding instant t63 at which the corresponding image 163, representing the 

25 same state of motion of the heart as the image 15 1 , was acquired. The image 163 can be used 
for subtraction angiography. The similarity comparison with the ECG E7 yields a 
corresponding instant t75. However, no image was acquired at that instant as is illustrated by 
the dashed arrow at the instant t75. Prior to the corresponding instant t75 the image 172 was 
acquired and the image 173 was acquired after the instant t75. 

30 A first possibility of obtaining an image which is suitable for subtraction 

angiography is to select that image whose acquisition instant lies as closely as possible to the 
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corresponding instant t75. In other words, that image is selected for which the time difference 
between the acquisition instant of the image and the corresponding instant t75 is smallest. 
This would be the image 173. This possibility is used notably when there are further image 
sequences with ECGs (not shown) which have been acquired without using a contrast 
5 medium. For each of these further ECGs this method enables the determination of a time 
difference between the relevant corresponding instant and the nearest image. The image for 
which the time difference is smallest is chosen for subtraction angiography. 

Another possibility for obtaining an image which is suitable for subtraction 
angiography is to interpolate an image from those images which have been acquired each 

10 time before and after the corresponding instant. For example, the motion of the heart between 
the states of motion shown in the images 172 and 173 can be interpolated so as to form an 
artificial image which represents the state of motion of the heart at the corresponding instant 
t75. In a simple case a linear motion of the heart is assumed between the states of motion of 
the images 172 and 173. In another case the overall motion of the heart is described in a 

1 5 motion model whereby the almost exact motion of the heart between the acquisition instants 
of the images 172 and 173 can be interpolated and an image of the state of motion at the 
corresponding instant t75 can be determined. 

The use of the method in accordance with the invention is not limited to one 
reference image. In the case of an image sequence comprising a plurality of reference images, 

20 a corresponding image can be determined or formed for each individual reference image by 
means of the method. 

Fig. 8 is a diagrammatic representation of a medical X-ray examination 
system. The system includes an X-ray source 40 which is arranged to emit X-rays 42 in such 
a manner that they traverse an object to be examined, in this case being a patient 41 who is 

25 arranged on a table 43 which is transparent to X-rays; subsequently, the X-rays can be 

detected by an X-ray image detector 44 which is arranged underneath the table 43. The X-ray 
image detector 44 comprises an array of sensor elements which are sensitive to X-rays. 
Because of the organ-specific attenuation of the X-rays 42 while traversing the patient 41, an 
image is formed in the X-ray image detector 44, the data of said image being applied to a 

30 data processing unit 46. The data processing unit 46, also being capable of executing system 
control tasks which are not described herein, processes the incoming image data 45 in such a 
manner that an optimum image is formed for a viewer. The image data 47 thus processed is 
applied to a visualization unit 48, for example, a monitor, on which the data can be presented 
to a viewer. The data processing unit can optionally be constructed so as to be programmable. 
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In that case a data reading apparatus 52 is available which is coupled to the data processing 
unit 46 and is capable of reading a computer program from a computer program product so as 
to apply it to the data processing unit 46. The computer program enables the data processing 
unit inter alia to carry out the method in accordance with the invention. 

Electrodes which are connected to an ECG apparatus 51 are attached to the 
patient. The Figure shows, merely by example, only one connection lead 50 with an electrode 
attached to the body. In reality a plurality of electrodes will be used in known manner as well 
as an electrode (not shown) for a reference potential (often ground). The ECG apparatus 
connected to the data processing unit 46 forms the ECG of the patient 41 during the X-ray 
image acquisition under the control of the data processing unit 46, and presents it to the data 
processing unit 46 in order to carry out the method in accordance with the invention. 

Also shown is a catheter 49 which is typically introduced into a blood vessel 
in the region of the groin of the patient during cardiological examinations. The physician then 
advances the tip of the catheter as far as the heart while acquiring X-ray image sequences 
with a low dose which serve as an aid for the navigation within the body. Once the heart is 
reached, via the catheter contrast medium is injected into the blood vessels of the heart. 
Briefly before the emergence of the contrast medium from the tip of the catheter the X-ray 
image detector 44 and the X-ray source 40 are switched to a high-dose mode of operation for 
the subsequent acquisitions, so that detailed high-dose images are formed of the vascular tree 
of the heart which is filled with the contrast medium. 

Subsequently, the X-ray source is deactivated or low-dose images are formed 
again should the physician require navigation aids for further actions. The high-dose images 
of the vascular tree filled with the contrast medium are stored in the data processing unit 46. 
Using the ECG formed by the ECG apparatus 41 and the method in accordance with the 
invention, these high-dose images can be superposed in the described manner on the low- 
dose images or on further high-dose images, representing the vascular tree without contrast 
medium, so as to be presented to the physician by way of the visualization unit 48. 

The method in accordance with the invention can be used for various 
cardiological examinations. 

1 . A plurality of cardiac cycles of the beating heart is acquired without injection of a contrast 
medium. Subsequently, a few cardiac cycles are acquired, or only a single cardiac cycle, with 
a contrast medium and subsequently a few cardiac cycles again without the contrast medium. 
After a respective image without contrast medium has been determined for several or all 
images of a cardiac cycle, which images represent the heart filled with the contrast medium, 
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by means of the method in accordance with the invention, and after its subtraction therefrom, 
there is obtained the image sequence of a cardiac cycle in which the images represent 
substantially exclusively the vascular tree of the heart in the various states of motion of a 
cardiac cycle. These images are stored in a data processing unit and can be repeatedly 
5 presented to a physician as a moving image sequence. The images with the contrast medium 
then constitute the reference images and the images without the contrast medium form the 
corresponding images. 

2. The data processing unit of an examination system contains an image sequence as formed 
in the above section 1, its images representing practically exclusively the vascular tree of the 

1 0 heart in the various states of motion of a cardiac cycle. In order to position a catheter near the 
heart, the physician utilizes an aid for navigation in the body in the form of X-ray image 
sequences which are acquired continuously with a low dose. In these sequences the vascular 
tree and the heart can be recognized only with difficulty because of the physical conditions. 
For an X-ray image just acquired a corresponding image is automatically or manually 

15 determined from the stored sequence so as to be superposed on the X-ray image by means of 
the method in accordance with the invention. As a result, the vascular tree filled with the 
contrast medium is superposed on the instantaneous X-ray image so that the physician is 
offered a suitable navigation aid. If desired, such superposition can also be performed 
continuously for practically every acquired X-ray image. The images without the contrast 

20 medium then form the reference images and the images with the contrast medium form the 
corresponding images. 

It is to be noted that the method is not restricted to X-ray examination systems. 
The method in accordance with the invention can be carried out in an ultrasound examination 
system in the same way as in the X-ray examination apparatus shown in Fig. 8. To this end, 

25 the reference images and the corresponding images are formed by ultrasound, an ultrasound- 
reflecting contrast medium being administered to the vessels or the tissue to be examined 
either during the acquisition of the reference images or of the corresponding images. 



